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Swarm verification and parallel randomised depth-first search are very effective parallel techniques 
to hunt bugs in large state spaces. In case bugs are absent, however, scalability of the parallelisation 
is completely lost. In recent work, we proposed a mechanism to inform the workers which parts of 
the state space to explore. This mechanism is compatible with any action-based formalism, where a 
state space can be represented by a labelled transition system. With this extension, each worker can 
be strictly bounded to explore only a small fraction of the state space at a time. In this paper, we 
present the Hive tool together with two search algorithms which were added to the LTSmin tool 
suite to both perform a preprocessing step, and execute a bounded worker search. The new tool is 
used to coordinate informed swarm explorations, and the two new LTSmin algorithms are employed 
for preprocessing a model and performing the individual searches. 



1 Introduction 

In explicit-state model checking (MC), it is checked whether a given system specification yields a given 
temporal property. This is done by exploring the so-called state space of the specification, which is a 
directed graph describing explicitly all potential behaviour of the system. Since state space exploration 
algorithms often need to keep track of all explored states in order to efficiently perform the MC taskQ 
and since state spaces can be very large, for many years, the amount of available memory in a computer 
has been the most important bottleneck for MC. 

In recent years, however, the increase of available memory in state-of-the-art computers has con- 
tinued to follow Moore's Law [12], while the increase of their processors' speed no longer has. For 
MC, this means that large state spaces can be stored in memory, but the time needed to explore them is 
impractically long, hence a time explosion problem has emerged. This can be mitigated by developing 
distributed exploration algorithms, in which a number of computers in a cluster or grid are used to per- 
form an exploration. Many of those algorithms use a partitioning function to assign states to workers, 
and require frequent synchronisation between these workers, see e.g. ifTl l2l 141171 [TOl . 

Swarm verification [9] (SV) (and parallel randomised Depth-First Search [5]) are recent techniques 
to perform state space exploration in a so-called embarrassingly parallel (6) way, where the individual 
workers never need to synchronise with each other. In SV, each worker starts at the initial state and 
performs a search based on Depth-First Search (Dfs). The direction of a worker is determined by a 
given successor ordering strategy. As the direction of a Dfs depends on the fact that a stack is used to 
order successor states (i.e. a Last-In-First-Out strategy), changing this ordering directly influences the 
direction of the search. By providing each worker a unique strategy, they will explore different parts of 
the state space first. With this method, some states may be explored multiple times by different workers, 
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Table 1: The four major functionalities of ISV 



LTSmin 


Hive 


PI. Trace-counting Drs: Constructs CP' with tc{s) = min(l,£ J / eN /Ic(i')). 
F2. Informed Swarm Search (Iss): Search of 9 restricted to a 


Fl. Trace selection: select a swarm trace a for worker 
F3. Update swarm set: remove inspected traces 



but if the property does not hold, any bug states present are likely to be detected very quickly, due to the 
diversity of the searches, which often means that the whole state space does not have to be explored. 

However, if a property holds, each worker will exhaustively explore the whole reachable state space, 
which means that the benefits of parallelisation are completely lost. Recently, we proposed a mechanism 
to bound each worker to a particular reachable strict subset of the set of reachable states, in such a way 
that together, the workers explore the whole state space Ifl4l . This mechanism is compatible with any 
action-based formalism such as /iCRL (H, where each transition in a state space is labelled with some 
action name corresponding with system behaviour. In this paper, we explain how the Heuristics Instruc- 
tor for parallel VErification (Hive) tool, which resulted from Ifl4l . works in practice. Section[2]presents 
the functionality of the Hive tool together with some new algorithms implemented in the LTSmin tool 
suite [4j . How all these have been implemented and how the resulting tools can be used is explained in 
Section[3] In Section|4| experimental results are discussed. Finally, conclusions and pointers for possible 
future work are given in Section [5] 

2 The Informed Swarm Exploration Technique 

The Setting The so-called Informed SV technique (ISV) implemented in Hive and LTSmin is appli- 
cable if three conditions are met: (1) A system specification *p should be an implicit description of a 
Labelled Transition System (Lts) 7. An Lts 7 is a quadruple (§,A,7,Sj n ), where S[ n is the initial state, 
S is the set of states reachable from Sj n , A is a set of transition labels (actions), and 7 : § x A x S is the set 
of transitions between states. With s —It, A C A, we say that there exists an £ G A such that (s, £, t) G 7. 
The reflexive transitive closure of — > is denoted as — >* . In on-the-fly state space exploration, s,„ and A 
are known a priori, but S and 7 are not, and a next-state function Jsf : § — > 2 s provides the set of successors 

A 

of a given state. A state t is the successor of a state s iff s — > t. Ji is used to construct S and T, starting at 
Si n . In the following, we use the notation N | A, with A C A, to denote Tsf restricted to a set of transition 
labels A, i.e. N | A(s) = {s' G S | 3£ G A.(s,£, s') G 7}. Clearly, N | A = N. Finally, a sequence of actions 

(£o,£i, ■ ■ .) describes all transition sequences (traces) through an LTS T with s,- n ^-H^o ^Xsi • • • for some 
s§,s\ , . . .. If CP is label-deterministic, i.e. for all s, t G S, £ G A with s — > t, there does not exist a state 
t' 7^ t with s — > t', such an action sequence corresponds to a single trace. Here, we assume that all LTSs 
are label-deterministic. If this is not the case, relabelling of some transitions can resolve this. 

(2) *p should consist of a finite number n > 1 of process descriptions (e.g. process algebraic terms) 
in parallel composition. This is the case for any concurrent system. (3) At least some of these processes 
in parallel composition, i.e. a subsystem, should yield finite behaviour, hence only finite traces. This is 
not a strict requirement, but if it is not met, then the method relies on bounded analysis of the subsystem, 
and it does not automatically guarantee anymore that all reachable states are visited. 

ISV Say that a specification describes a system of concurrent processes *}3 = {Pq, . . . ,P n }, with n G IN. 
ISV exploits the fact that parallel composition is a major cause for state space explosion, and that LTS 
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y of *p is the synchronous product of LTSs 3 ,- = (S'j-A'jT", J- n ) of the Z 5 ; (0 < i < n), restricted by some 
synchronisation rules between processes, given by a symmetric function £ E.g. &(£,£') = £" states that 
if actions £ and £' can be performed by different processes, then the result is action £" in the system. 
For the formal details, see [14]. We assume that the A' are disjoint (if this is not the case, then some 
rewriting can resolve this^ and that no action is involved in more than one rule defined by £ (either 
as an input, or as a result). All this implies that for any I G A, it can be determined whether or not it 
stems from some behaviour of a particular process Pj. Say that A c C A is the set of actions stemming 
from synchronisations, and that A' c C A 1 is the set of actions of IP,- which are forced to synchronise with 
other actions, then A = {\J i<n A' \A[) U A c . Now, for any A C A, we can define 9Jt(A) as {£" G A c \ 
3£ G A,£' g" A.<£(£,£') =£!'}, which is the set of actions resulting from synchronisation involving one 
action in A. Finally, the assumptions about £ allow us to define a relabelling function 9\ as follows: 
m({£}) = {£"} iff there exists an £' such that &(£,£!) = £", and <R{{£}) = {£}, otherwise. 

Four basic functionalities are 
required to perform ISV in prac- 
tice. These are listed in Table [1} 
a preprocessing step (PI) involv- 
ing the analysis of a defined sub- 
system yielding finite behaviour, 
and three techniques for the three 
major phases of ISV (Fl-3). PI 
and F2 require two new search 
algorithms, which have been im- 
plemented in LTSmin. Fl and 
F3 have been implemented in 
the new stand-alone Hive tool. 
The general procedure to per- 
form an ISV is as follows: First, 
the user selects a strict subsys- 
tem *p' C which is guaranteed 
to yield finite behaviour (again, 
alternatively, this behaviour is 
bounded in the ISV, but then, 
the overall search could be non- 
exhaustive). In PI, the LTS IP' = 
(S', A',7',s' in ), described by is constructed and saved to disk, together with a weight function 
tc : §>' — > M. For this, we have extended the Dfs implementation in LTSmin, as described in Alg. [I] 
The tc function (see also Table [TJ assigns 1 to deadlock states, i.e. if N(s) = 0, and the sum of all the 
successor weights to any other state (note that 3sf' is the next-state function of 7'). 

This allows efficient reasoning about the traces through J"; the number of traces represented by a 
trace prefix from Sj„ to some jG§' equals tc(s). E.g., Fig. [I] shows a simplified acyclic LTs[^]of an 
iPod process as part of the DRM protocol specification from |[T3l . with part of the definition of tc. With 
this, if we sort the states based on their numbering (which was assigned by the Dfs), then each trace 
can be uniquely referred to with a natural number: note that the number of possible traces is 14, which 
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► [10,12) 
► [10,12) 
> [10,12) 

[10,11) 



Figure 1: The LTS of an iPod process, with weights, and trace nr. 10 



2 Strictly speaking, a weaker requirement suffices [ 14|. 

3 The actual LTS of this example, in which the actions are extended with some data parameters, consists of 547 states. 
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corresponds with tc(0). Say that we want to identify trace 10 (shown in Fig. [TJ. State has successors 
{1, . . . ,4}. Sorted by increasing state number, we first consider state 1; since tc[\) = 3, we conclude that 
trace (recv_Cl_3) represents traces to 2, i.e. 3 traces, starting at trace 0. 



IN are constructed 



We also have tc(2) = 3, therefore (recv_Cl_l) represents 
traces 3 to 5. Similarly, (recv_Cl_2) represents traces 6 to 12, 
and (off(Cl)) represents trace 13. This means that (recv_Cl_2) 
is a prefix of trace 10. Since state 3 only has state 5 as a succes- 
sor, clearly (recv_Cl_2,send) is also a prefix (this agrees with 
tc(5) = 7: all 7 traces represented by (recv_Cl_2) are also rep- 
resented by (recv_Cl_2,send)). In this fashion, the complete 
trace can be constructed following the states listed on the right of 
Fig. [T] This principle is used for trace selection in Hive (Fl). In 
ISV, each worker is bounded by a trace through given by Hive 
(this will be explained next). Therefore, each trace represents a 
worker job to be performed, and 7' represents the set of jobs. 
From a trace (£q, ...,£„) through T' (n £ B), a so-called swarm 

trace o = (9\(£o), . . . ,${(£„)) can be constructed, taking into account synchronisations with ?fi\ s $'. 
Whenever a worker thread can be launched, Hive selects a swarm trace. When the Hive tool is launched 
to start an ISV, this is done first. 

A launched worker thread performs an informed swarm search (ISS), implemented in LTSmin 
(Alg. [2] and F2). In Alg. [2j a is the swarm trace assigned by the Hive tool, and a(i) is the singleton 
set containing the (i+ 1) element of o (If a contains fewer than i+ 1 elements, we say that a(i) = 0). 
In the ISS, "? is explored, but not exhaustively: the potential behaviour of the subsystem is restricted 
to a, which restricts exploration of 7. For each visited state s, Next is extended with N | (,A\A)(j), 
i.e. all successor states reachable via behaviour of *p\*p', and Step is extended with N | a(i)(s), i.e. all 
successor states reachable via the current behaviour in a. 



Algorithm 1 Trace-counting Dfs 

Require: <p' C <p, s' in , A' 
Ensure: J" and tc ; S' 

Closed <- 

dfs(s) = 

if s g Closed then 
fc(.v) <- 

for alii' e N'(s) do 

tc(s) i- tc(s)+dfs(s' 
if = 0then 

tc(s) «- 1 
Closed <— Closed U {s} 
return tc(s) 



V) 



When all states in Open are explored, the contents of Next is 
moved to Open, after duplicate detection (for which the search his- 
tory Closed is used). Note that when all reachable states have been 
explored in this manner, i is increased, by which the ISS moves to 
the next step in a, and new states become available. The main idea 
of ISV is to construct the set of all possible traces through J", and to 
perform an ISS through 7 for each of those traces. This means that 
eventually T is completely explored. A proof of correctness can be 
sketched as follows: say that all traces through 7' have been used by 
workers to explore 3\ and that after this, some reachable state sG§ 
has never been visited. We will show that this leads to a contradic- 
tion. It follows from Alg. [2] that for each state t to be explored, all 
new states t' € N | (A\A) (t) are going to be explored as well, and for 
some i, t" G N | a(i)(t) is going to be added to Step. This implies that all states i € N | (A \c{i))(t) are 
going to be ignored. From this and the fact that is explored, it follows that a state s is ignored iff for all 

traces through 7 from s,„ to s, there exist t,t G S such that Sj n t^-ti—t* s, with £ G A\a(i), i being 
the current position in a when exploring t. Let us consider one of those traces. We call a' the swarm 
trace followed to reach t from Sj n over that trace. Note that this is a prefix of a. Let us assume that by 



Algorithm 2 BFS-based ISS 

Require: t p, .v,„, A, A = A' U SDT(X'), a 
Ensure: T restricted to a is explored 

Open <r- Sj„; Closed, Next. Step, Fj <- 
while Open HIV Step ^ do 
if Open = then 
i <- i + 1 

Open <— Step \ Closed; Step, F, 
for all .v e Open do 

Next <- Next U N | (A \ A ) (s ) 
Step <- StepUN a(i)(s) 
Fi^FiU{£eA\ X\ {£}{s)^Q} 
Closed 4- Closed U Open 
Open «- Next\ Closed; Next <- 
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following a' extended with £, s can be reached from Sj n £j Since a' has been derived from a trace through 
IP' and I E A, the extended trace must also be derivable from a trace through J 3 '. But then, since all traces 
through IP' have been used in the ISV, s must have been visited by some other worker that followed a', 
and we have a contradiction. 

In case *p and ty' sometimes synchronise, the trace counting will produce an over-approximation of 
the possible set of traces of *p' in the context of This is because in the trace counting, it is always 
assumed that whenever needs to synchronise, this can happen in The result is that some swarm 
traces may not correspond with actual potential behaviour in IP. To deal with this, ISS includes a feedback 
procedure: For every position i in a, it is recorded in F, which potential behaviour of *p' has actually 
been observed. When finished, ISS returns the F,-, and using these, Hive can prune away both a and 
other, invalid, traces (F3). Since each trace prefix represents a set of traces with consecutive numbers 
(see e.g. the ranges for states in trace 10 in Fig. [TJ, the set of explored and pruned swarm traces can 
be represented in a relatively small list of ranges. Initially, the set of swarm traces is empty. Say we 
explore the Lts IP of the DRM specification, and IP' is as displayed in Fig. [I] and say it is detected 
that synchronisation with recv_Cl_2 at state (to state 3) can actually not happen in IP. As already 
mentioned, (recv_Cl_2) represents [6, 13). So after pruning, [6, 13) represents the new set of explored 
traces. Furthermore, elements in the list can often be merged. E.g., if ranges [0,5) and [8, 14) have been 
explored earlier, and range [5,8) is to be added, the result can again be described using a single range 
[0, 14). At all times, Hive is ready to launch another worker (Fl) and to prune more traces (F3). When 
there are no more swarm traces left to explore, the ISV is finished. 



3 Implementation and Using the Tools 

Implementation The trace-counting Dfs and ISS have both been implemented in an unofficial exten- 
sion of the LTSmin toolset version 1.6-19, which has been written in C. Since LTSmin already contains 
a whole range of exploration algorithms (both explicit-state and symbolic), there was no need to imple- 
ment new data structures. ISV is very light-weight in terms of communication between the Hive and the 
workers, the only information sent to launch an ISS being a swarm trace in the form of a list of actions. 
This list is being stored in LTSmin in a linked list, and a pointer traverses this list when exploring, to 
keep track of the current swarm trace position. In addition to this, a bit set is used to keep track of 
the encountered actions stemming from *p' since the last move along the swarm trace. This is done to 
construct the Fj. A bit set implementation using a tree data structure is available in the LTSmin toolset. 

Unfortunately, it is currently not possible to automatically extract ty' of a given subsystem from *p, 
meaning that ty' must manually be derived from *p. At times, this requires quite some inside knowledge 
of the description, therefore it is at the moment the main reason that we have not yet performed more 
experiments. Automatic construction of the *p' description is listed as future work (see Section]?]). 

The Hive tool consists of about 1 ,200 lines of C-code. Because of the communication being light- 
weight, and because interactions between the workers and Hive either involve asking for a new trace 
and receiving it, or sending the results of an ISS, we decided to implement all communication in the 
request-response method using TCP/IP sockets. During an ISV, Hive frequently needs to extract traces 
from the IP', which is kept in memory together with the fc-function. Besides this, a linked list L of nodes 
containing trace ranges (the [i,j) mentioned in Section[2]) is maintained, representing the set of explored 



4 This is not true if there are multiple transitions stemming from ?p' on the trace to s not agreeing with a, but then, we can 
repeat the reasoning in the proof sketch until there is only one left. 
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traces. Currently, when a trace is selected (Fl), an ID is chosen randomly from L, but one can imagine 
other selection strategies (see Section[5]). Then, the corresponding trace is extracted from V. 

When launching many workers, frequent requests to Hive are to be expected. Therefore, Hive has 
been implemented with pthreads; whenever a worker sends a request, a new thread is launched in Hive 
to handle the request. If a new swarm trace is required, Fl is performed, and if feedback is given, F3 is 
performed. The Lts 7' is never changed, hence no race conditions can occur when multiple threads read 
it, but L is frequently accessed and updated, when selecting a trace and processing feedback, the latter 
involving writing. For this reason, we introduced a data lock on L. We plan to use more fine-grained 
locking in the future, but we have not yet experienced a real slowdown when using one lock. 

During an ISV, Hive keeps accepting new requests until L has one node containing the range 
[0, tc(s' in )). From that moment on, any requests are answered with the command to terminate, effectively 
ending all worker executions. The same is done if a worker reports in its feedback that a counter-example 
to a property to check has been found, because the ISV can stop immediately in that case. 

Finally, all has been tested on Linux (Red Hat 4.3.2-7 and Debian 6.0.1) and Mac OS X 10.6.8. 

Setting up and launching an ISV In the following, we assume that we have a p,CRL specification 
named spec.mcrl describing and a specification named specsub.mcrl describing Actually, any 
action-based modelling language compatible with LTSmin is suitable for ISV as well. A /^CRL spec- 
ification is usually first linearised to a tbf file, using the /xCRL toolset [3], which is subsequently used 
as the actual input of LTSmin. Having spec. tbf and specsub.tbf, the weighted V is saved to disk as 
follows, with (sub) being the chosen base name for the files storing the weighted J" j^] 

lpo2lts-grey -getswarm=(sub) specsub.tbf 

This produces the files sub.swh, sub.swc, and sub.sww, containing the actions in A', the transitions in 
7' (with actions and states represented by numbers), and the weights of the states, respectively. Actually, 
if specsub.tbf yields infinite behaviour, this can be bounded by a depth « using the option -swbound=(n). 

The Hive can now be launched on the same machine by invoking the following, with (portnr) being 
the port number it is supposed to listen at for incoming requests: 

hive (portnr) (sub) 

An ISS can be started as follows, (server) being the IP address of the machine running Hive: 

lpo2lts-grey -swarm=(sub) -hiveserver= (server) -hiveport=(port) spec. tbf 

Note that each ISS also needs information on V . Actually, only sub.swh is read from disk, to learn 
A'. Therefore, this file should be available on all machines where Is Si are started. Finally, in practice, 
one often wants to start many ISSs simultaneously, and start a new ISS every time one terminates. This 
whole procedure can be launched using the shell script hive_launch.sh. 

4 Experimental Results 

Table [2] shows experimental results using the /iCRL [8] specifications of a DRM procotol lfT3l and the 
Link Layer Protocol of the IEEE-1394 Serial Bus (Firewire) [11]. We were not yet able to perform 



5 For /xCRL specifications, lpo2lts-grey is the explicit-state space generator of LTSMIN. For other modelling languages, 
another appropriate LTSMIN tool should be used. 
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Table 2: Results for bug-free cases with SV and ISV, 10 and 100 workers. 



case 


# workers 


search 






results 












# est. runs 


# runs 


max. # states 


max. time 


total # states 


total time 




10 


SV 


10 


10 


13,246,976 


19 ,477 s 


132,469,760 


19,471 s 


DRM (lnnc, 3ttp) 


10 
10 


ISV, 1 iPod 
ISV, 2 iPods 


5,124 
1.31* 10 13 


45 
7,070 


2,352,315 
70,211 


2,832 s 
177 s 


85,966,540 
353,591,910 


14,157 s 
125,139 s 




100 


ISV, 2 iPods 


1.31* 10 13 


9,900 


70,211 


175 s 


361,050,900 


17,325 s 




10 


SV 


10 


10 


137,935,402 


105,020 s 


1,379,354,020 


105,020 s 


1394 (3 link ent.) 


10 


ISV 


3.01* 10 9 


1,160 


236,823 


524 s 


235,114,520 


60,784 s 




100 


ISV 


3.01* 10 9 


1,400 


236,823 


521 s 


252,206,430 


7,294 s 



# est. runs: estimated # runs needed (# of swarm traces for ISV). # runs: actual # runs needed, total (max.) # states: total 
(largest) # states explored (in a single search), total (max.) time: total (longest) running time (of a single search). 



more experiments using other specifications, mainly because subsystem specifications still need to be 
constructed manually, which requires a deep understanding of the system specifications. The experiments 
were performed on a machine with two dual-core AMD OPTERON (tm) processors 885 2.6 GHz, 126 GB 
RAM, running Red Hat 4.3.2-7. We simulated the presence of 10 and 100 workers for each experiment 
(the fully independent worker threads can also be run in sequence). This has some effect on the results: 
in order to simulate 10 and 100 parallel ISSs, HlVE postponed the processing of ISS feedback until 10 
and 100 of them had been accumulated, respectively. When the ISSs truly run in parallel, this feedback 
processing is done continuously, and redundant work can therefore be avoided at an earlier stage. In the 
DRM case, we selected both one and two iPod processes for ^3', and in the Firewire case, a bounded 
analysis of one of the link protocol entities resulted in IP'. The SV runs have been performed with the 
Dfs of LTSmin. Since the specifications are correct, there is no early termination for the explorations, 
meaning that in SV, all reachable states are explored 10 times. In the DRM case, ISV based on one iPod 
process leads to an initial swarm set with 5,124 traces, 45 of which were actually needed for different 
runs. Each run needed to explore at most 18% of 7, and in total, the number of states explored was 
smaller than in the SV. ISV based on two iPod processes leads to a much larger swarm set, and clearly, 
feedback information is essential. ISV with 10 parallel workers explored in total 2.5 times more states 
than SV, but each ISS covered at most \% of J\ meaning that they needed a small amount of memory. 
This demonstrates that ISV is useful in a network where the machines do not have large amounts of 
RAM. In the Firewire case with 10 parallel workers, each ISS explored at most only g% of 3\ and in 
total, the SV explored 83% more states than the ISV. In terms of scalability related to the number of 
parallel workers, the results with 100 workers show that the overall execution times can be drastically 
reduced when increasing the number of workers: compared to having 10 workers, 100 workers reduce 
the time by 86% in the DRM case, and 88% in the Firewire case. The number of ISSs has actually 
increased, but we expect this to be an effect of the simulations of parallel workers, as explained before. 

A full experimental analysis of the algorithms would also incorporate cases with bugs, to test the 
speed of detection. This is future work, but since ISV has practically no overhead compared to SV, 
and the ISSs are embarrassingly parallel and explore very different parts of a state space, we expect 
ISV and SV to be comparable in their bug-hunting capabilities. Finally, we chose not to compare ISV 
experimentally with other distributed techniques (e.g. those using frequent synchronisations), because 
there are too many undesired factors playing a role when doing that (e.g. implementation language, 
modelling language, level of expertise of the user with the model checker). 
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5 Conclusions and Future Work 

We presented the functionality of the Hive tool and two new LTSmin algorithms for IS Vs. ISV is an 
SV method for action-based formalisms to bound the embarrassingly parallel workers to different Lts 
parts. Worst case, if the system under verification is correct, no worker needs to perform an exhaustive 
exploration, and memory and time requirements for a single worker can remain low. 



Tool availability Both the ISV extended version of LTSmin and Hive are available at http:// 
|www . win . tue . nl/~aw i j s/suppls/hive_ltsmin . html . 

Future work We plan to further develop the Hive tool such that a description of a given sub- 
system can automatically be derived from a given description ^. We also wish to investigate which 
kind of subsystems are particulary effective for the work distribution in ISV, and which are not, so that 
an automatic subsystem selection method can be derived. If yields infinite behaviour, we want to 
support its full behaviour automatically in the future. As long as 3> is finite-state, this should be possible. 
Furthermore, we want to investigate different strategies to select swarm traces and to guide individual 
ISSs. Finally, we will perform more experiments with much larger state spaces, using a computer cluster. 
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